1. Field of the Invention
This invention relates to a technology for making a three-dimensional shape model (i.e., computer graphics model) with a computer and the like, and more particularly a three-dimensional model making device and method for making a three-dimensional shape model of an object to be seen as a synthetic image combined with an acutal image, i.e., relates to a special effects model making device and method for utilizing information from a single frame picture image and for generating and superimposing a three-dimensional shape model over actual images.
2. Description of Related Art
As a method for making a three-dimensional shape model of a three-dimensional object with a computer and the like, there is provided a method in which a user uses a graphical user interface (GUI) and the like to input either a shape of the object or its coordinates in an interactive mode. As methods for inputting a shape or coordinates, there can be provided a method in which a three-plane figure is applied, a method in which a figure most suitable for an object to be made is selected from a group of basic figures called primitives which are properly modified to make a model, and a method in which lengths or coordinates are directly inputted using numerical values and the like. For example, the gazette of Japanese Patent Laid-Open No. Hei 5-342310 with a title of "Three-Dimensional Converting Device For Line Element Data And Its Method" discloses a device capable of drawing a sketch on a paper sheet when a three-dimensional object is inputted.
Various situations or cases may exist in model making, i.e., as conditions for making a model of an object, there may be provided: (1) a case in which a shape of the object is already known or can be measured; (2) a case in which a shape of the object is not yet known or defined and a model is made while determining either a practical shape or a practical size; and, (3) a case in which a shape of the object is not known, but its actual image is present. Accordingly, one may apply a method of making a model of an object of which a shape is not known while referring to its actual image, and to make such model in such a way that the model may approach a shape of the actual image as near as possible. However, since this method is merely applied at most to a system in which the actual image is applied only for a reference to make the model of the object, two-dimensional information concerning the shape of the object of the actual image is not reflected directly in the making of the model.
Gazette of Japanese Patent Laid-Open No. Hei 5-233781 with a title of "Cubic Model Making System With Animation Image" provides a method in which a cubic model is made while referring to an actual image, i.e., wherein an object within the actual image and a generated image of the cubic model are overlapped and displayed on a same screen. Within such case, a user can easily confirm a difference between the actual imaged object and a generated image of the cubic model. However, also in this case, the actual image is merely used in confirmation of the produced model and it does not refer to the utilization of information when the model is made. In addition, it is necessary that camera parameters of the actual image required for overlapping the actual image on the produced model be already known.
It is generally impossible to perform a direct making of a three-dimensional model from a two-dimensional information of an actual image. More particularly, since a depth information of an optional displayed point cannot be seen from two-dimensional image information, it is not possible to specify uniformly in what position is the depth information placed in view of its three-dimensional state. However, actually, it is possible to attain a specified position if various assumption conditions are specified and this process is called cubic acknowledgement.
Cubic acknowledgement can be roughly classified into two processes. One process is a method in which a plurality of images seen from different points of view are used, and the other process is a method in which information other than the two-dimensional coordinates contained in the image is utilized from only one image to recover the three-dimensional structure.
One method using a plurality of images is a cubic acknowledgement method with a stereo image. The same object is taken with two or more cameras installed at different locations, and its three-dimensional coordinates values are calculated in reference to information of relative positional relation of each of the cameras and two-dimensional coordinates values on each of the images of the point seen on the image.
Several methods for acknowledging a cubic with one image are as follows, i.e., (1) a method for referencing a point where the three-dimensional coordinates values are already known; (2) a method for comparing an image with a model of a basic shape prepared in advance; and, (3) a method of using texture information of an image and the like.
More particularly, discussing the above method (1) in greater detail, if there are more than three points having already-known three-dimensional coordinates values, it is possible to obtain three-dimensional coordinates with a camera eye and a direction of a sight line. This problem is called in general as PnP (Perspective-n-Point Problem) (Kohichiro Deguchi: "Image and Space", Syokohdoh Publishing, 1991). However, this method is merely operated to calculate data of each of cameras at the most and cannot obtain three-dimensional coordinates of other points contained in the image.
Discussing the above method (2), the method of comparing the object seen on the image with the model of basic shape prepared in advance is carried out such that three-dimensional models of various kinds of basic shapes are prepared in advance, the images seen through a perspective conversion are compared with the actual images so as to estimate and approximate the three-dimensional model.
As a system in which the present method is applied to the three-dimensional model formation, there is a gazette of Japanese Patent Laid-Open No. Hei 4-289976 with a title of "Method For Making A Three-Dimensional Shape Model And Its System". Such provides a method in which a two-dimensional figure information of a three-dimensional object to be processed is inputted, a three-dimensional basic shape model of which a basic shape is approximate to that of the object is inputted, and their control points are corresponded to each other so as to convert the basic shape model and further to make a three-dimensional model.
Discussing the above method (3), as a typical method using the texture information of an image, there is provided a Shape From Texture Process. An orientation of a plane of a surface of the object is calculated in reference to a rich or a lean texture concentration in the surface of the object. There is a disadvantage that an accuracy in calculation may depend upon the kind of texture at the surface of the object or the like.
In the method in which a three-dimensional model is made without applying any direct use of actual image and the formed model is displayed while being overlapped on the actual image so as to confirm the formed model when the three-dimension the object seen on the actual image is formed, the shape information of the object contained in the actual image is not effectively utilized when the model is made. Due to this fact, in a case where the model corresponding to the object of the actual image is made when the three-dimensional model is formed, it is necessary to repeat a modelling and its confirmation numerous times on a trial-and-error basis.
In the method in which the three-dimensional basic shape model is inputted and the model is deformed in such a way that the object of the actual image and its model may be coincided to each other in two-dimensional images, it is necessary that parameters such as a position and an orientation of the camera taking that image or the like be already known, and further, it is necessary that control points of the basic shape model and the object on the actual image are corresponded to each other.
In the method using a plurality of images (stereo-scope), it is necessary to prepare a plurality of images in which the same object is taken from different directions.
While the above-described methods provide some advantages and virtues regarding three-dimensional modelling, there still exists a need for an improved three-dimensional modelling method.